Decision Boundary Formation of Neural Networks

In this paper, we provide a thorough analysis of decision boundaries of neural networks when they are used as a classifier. First, we divide the classifying mechanism of the neural network into two parts: dimension expansion by hidden neurons and linear decision boundary formation by output neurons. In this paradigm, the input data is first warped into a higher dimensional space by the hidden neurons and the output neurons draw linear decision boundaries in the expanded space (hidden neuron space). We also found that the decision boundaries in the hidden neuron space are not completely independent. This dependency of decision boundaries is extended to multiclass problems, providing a valuable insight into formation of decision boundaries in the hidden neuron space. This analysis provides a new understanding of how neural networks construct complex decision boundaries and explains how different sets of weights may provide similar results. Key-Words: neural networks, analysis of decision boundary, dimension expansion, linear boundary, dependent decision boundary. 1 The Korea Science and Engineering Foundation partly supported the publication of this paper through BERC at Yonsei University.